Hadoop on Windows with Eclipse

1. Introduction
2. Prerequistes
3. Set Environment Variables
4. Setup SSH daemon
5. Download hadoop and place it in the home directory.
6. Unpack hadoop
7. Configure Hadoop
8. Format the namenode
9. Setup hadoop plugin
10. Start the cluster
11. Setup hadoop location
12. Upload data
13. Create and run a test project.

Upload data to HDFS

We are now ready to run the first Map/Reduce project but data is still missing. This section explains how to upload data to the Hadoop Distributed File System (HDFS).

Upload Files To HDFS

1. Open a new CYGWIN command window.

   

2. Execute the following commands in the new CYGWIN window as shown on the image above.
   

   cd hadoop-0.19.1
   bin/hadoop fs -mkdir In
   bin/hadoop fs -put *.txt In

   When the last of the above commands starts executing, you should see some activity in other Hadoop windows as shown in the image below.

   
   

   The result of these commands is a newly created directory -- named In -- in the HDFS which contains a set of text files that comes with the Hadoop distribution.

3. Close the Cygwin Window.

Verify if the files were uploaded correctly

In this section we will check if the files were uploaded correctly.

1. Open the Eclipse environment.
2. Open DFS locations folder which is located in the Project Explorer tab of Map/Reduce perspective.
3. Open localhost folder in DFS locations folder.
4. Keep opening HDFS folders until you navigate to the newly created In directory, as shown in the image below.

    

 



Verifying that the data was uploaded correctly




5. When you get to the In directory, double-click on the file LICENCE.txt to open it.
6. If you see something similar to the image above then the data was uploaded correctly and you can proceed to your first Hadoop project.

 

You can now move on to the next step.

 

Continue